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Outline asa 


¢ Introduction 
— Seeking greater efficiency & performance through experiment design 
¢ Efficiency gained by collecting the “right amount” of data 
¢ Performance gained by adding statistical rigor 
¢ System Identification Process in Wind Tunnel 
— Design of Experiment (DOE) 
— Learn-to-Fly (L2F) 
— Blended DOE-L2F 
¢ First time testing blended concept — strawman approach 
¢ Work in progress 
¢ Analysis, Results, and Validation Tests 
— DOE Tests 
— L2F Tests 
— Blended DOE-L2F Tests 


¢ Concluding Remarks 
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Motivation: Seek Efficiency Using Experiment Design asa 


¢ Wide spectrum of modeling demands 
— Fidelity requirements 
— Aircraft complexity 


¢ Aircraft complexity drive up costs 
— Conventional practice in LaRC 12-foot Wind Tunnel (static test) 
¢ 100 Hz sample rate, dwell for 10 seconds, average data 
¢ ~ 2 data pts/min 
— Simple factorial test for L-59 
¢ 9-Factors: a, 8, and 7 control surfaces 
¢ 29 = 512 test points => 4.26 hours 
— Reasonable data density often requires 59 => 16,276 hours (~8years)! 
¢ Investigators must tradeoff of cost vs fidelity/complexity 
— Define purpose of model and required fidelity. What is allowable error? 
— Asking for “best possible answer’ is not adequate 


— Speeding up the modeling process helps anywhere on spectrum 
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Aircraft System Identification Process 


Model Experiment 
Postulation Design 


ee measured data 


Data Compatibility 
Analysis 


input-output data 
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Model Structure 
Determination 
& 
Parameter and State 
Estimation 


Different Data Sets 


Model Validation 
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Test vehicle for Wind Tunnel Static Test 
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> L-59 Albatros 

> Czech military trainer 

> Low-cost off-the-shelf kit 

> 12.5% scale model 

> Sport application, RC actuators 
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Tenets of Design of Experiment (DOE) Sy 


Sequential testing proceeds only as model complexity requires 


Block 1 validation 


Block 2 validation 
Orthogonal regressors Replication defines system Block 3 validation 
are uncorrelated Replication \ noise/uncertainty 
Blocks of data, 
collected as needed, 
each followed by 
Orthogonality Randomization model validation 


Blocking reduces effects Randomization removes 
of known factors of no effects of unknown 
direct interest systematic errors 
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Block Designs & Supported Models 


¢ Full factorial design 


Y= Bot+ LB xA+D dD Byxjixj, +E i=1,2,...,k 
i Nae | 


¢ Face-centered design (FCD) 


VS Rot DB Ht LB FLU Bete FHI 2k 
i i ix j 


¢ Nested face-centered design 


y= By t+ DB x, +DByx +E > Bx,x;+ DBigX; +é bet 
1 1 1x] 1 
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Block 1, DOE Design Metrics (9-factors) Sy 


Block Type Design Terms 


Gnctusivey | | |i 2 si 
7: Fraction FCD Quadratic 


Maximum Variance Inflation Factor (VIF), 
reflects lack of orthogonality in design; desire < 10 


% Power reflects statistical power of design, 
manages type-2 error; desire 2 80 


is 


y=B)+>B ADB +L EB MX +E 12=1,2,...5k oe 
1 


.. | 
— Lod 


Validation Test Performed after each block of data 
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Block 2 added to create Nested FCD 


pf Ganetusivey | | 2, sn? 


“FractionFCD |1___|_156 9.64 
Nested FCD 312 | Quadratic | 22.41 


u*i yor] Wil 


y=Bot+ DB x; +B. x? +) > Buxx,+>B e+e PSL ick. Xe 
1 1 1x] 1 


Require optimized design points to reduce VIF 


NASA Langley Research Center 9 of 26 


Final DOE Design, 3-blocks 


Block Type Runs 
pf inclusive) || 2, Si 


/, Fraction FCD 156 | Quadratic 9.64 
Nested FCD 22.41 
366 


l-optimal block provides test points that minimize prediction error 


Validation Test Performed after each block of data 
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DOE Design for 3 blocks vasa 


= i EE CEE 34 


¢ FCD (black) 
¢ Nested FCD (red) 
32 rea] 32 . l-optimal (green) 


B:beta (deg) 


32 32 
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Stepwise Regression Modeling asa 


¢ Stepwise regression used to select model parameters 


y=fyt> Bx t+ > By? + DY Bx, + Bux t-te i=1,2,...,23 
i i iz j i 


¢ Primary metrics utilized for model selection: 
— Stepwise Regression significance level: 95% — 99% 
— Standard ANOVA table analysis 
— Lack of Fit (LOF) measure of model error relative to pure error 
— Standard deviation (fit error) 
— PRESS (prediction error sum of squares) 
— Coefficient of Variation (C.V.% = std. dev. / mean) 
— e,/ Cymax %; (e; = Cy, measured — Cy, predicted) -Goore 2) 
— R*, adjusted R? , predicted R? , (family of metrics) 
2 _ Variation explained | 


— ; O< R21 
total variation 
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Learn-to-Fly (L2F) Testing asa 


¢ L2F approach adapted to wind tunnel 
— General L2F approach is real-time global modeling of aerodynamics 
— Applicable to wind tunnel or flight testing 
— Continuous sampling during dynamic test 

¢ This study is a “quasi-static” test 
— Continuous sampling while sweeping target points slowly 
— Batch processing, stepwise regression 

¢ Key to efficiency: Wide-band orthogonal inputs 
— Higher bandwidth (HBW) inputs applied to control surfaces 
— Lower bandwidth (LBW) inputs apply to other factors 

¢ L2F experiment design 
— Test grid is setup for LBW factors 
— LBW trajectories form a nested “FCD-like” design 
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Learn-to-Fly (L2F) Trajectories 
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Blended DOE-L2F Testing (“quasi-static” test) asa 


¢ Use key “efficiency features” of both approaches 
— DOE: 4 tenets, sequential testing blocks of data, with validation tests 
— L2F: HBW design for factors that accept wide-band inputs 


¢ Blended design both simplifies and complicates final design 
— Simplifies 9-LBW experiment to a 2-LBW + 7-HBW experiment 
— Complicates evaluation of design metrics 


¢ Strawman blended design 
— Design for 9-LBW experiment ensure all factors are included in design 
— Keep statistical advantages and design metrics of DOE 
— Assume “extra” data between target points enhances modeling 
— Assume blended design is obtained by removing redundant a— targets 


— Blended designs require rig move slow enough to allow full sweep of 
controls at each a—£ target point 
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Blended DOE-L2F Design Metrics (9-factors) Sy 


Block Type 
Pd inefuded [target points [| 2 in 
[Linear + FI 


Nested FCD 


*Squared factors are aliased 


Some Lessons Learned: 
Fewer blocks required with continuous sampling 
Divide optimal blocks! 
4" block provided too much data for the blended design. 
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Blended DOE-L2F Trajectories asa 
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DOE Model (3 blocks) 


Model & Meas data, beta=0, faps=20, surfaces zero 


madal 
Oo meas data low aca 
Oo meas data mid ada 
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DOE Modeling Progression 


PRESS, Block 1 


¢ Block I (FCD) 02 
¢ 1%tin series of sequential tests 
¢ Case #2 — error budget satisfied ° 
¢ Case #3 — best model is cubic _ ,,... 
¢ Case #4 — minimum PRESS 0.05 
¢ Case #6 — minimum Std. Dev 


| case# | | 8 a ST 
8 


**ei/Camax % 


*residual e; = Cy_measured — Cy predicted, **Cymax = 1.22 


Model terms Linear+ | quadratic + cubic + cubic + 3FI cubic + cubic + cubic + 
O. 2FI (12 2FI (20 3FI (32 3 3FI (68 3F1 (81 3FI (128 
l 
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Validation Test, DOE Block 1 


0.08 — 


e Residuals vs Run 


om 1 * Block 1, %4FCD 
| 1, ¢ Cy, low arange 
are ee a ee tes ¢ Case #3 model 


e § fail 3% error 


0.02 — 


Residuals 


+3 % error 


-0.02 — 


+ 3% error test 
¢ 8 points failed 
¢ Qof 81 allowed 


Run Number 


Validation tests reveal true prediction & bias errors 


NASA Langley Research Center 20 of 26 


Validation Test, DOE Blocks 1-3 So 


¥ ¢ Residuals vs Run 
¢ Blocks 1-3, opt. 

° Cy low arange 

¢ Case #3 model 

e Similar final stats 
| 


Residuals 
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Model confirmed by validation test; 6 points fail 3% error test 
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Source of Cubic Terms (DOE blocks 1-3) aa 


Model Error (e/C,ymax) 
(a= 4°, dela_L = 12.5°) 
¢ Case #1, Linear — 4.78% 
° Case #2, Quadratic — 1.52% 
¢ Case #3, Cubic — 0.13% 


-25 -15 -5 5 15 25 
dela_L, deg 
NASA Langley Research Center 22 of 26 


L2F Test in LaRC 12-Foot Tunnel 
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Validation Test, L2F asa 


¢ Residuals vs Run 


_ ¢ Block L2F 
004 ° Cy, low arange 
bit | Fi - ¢ Case #3 model 
0.02 th WARS ‘ ‘ ‘ 
; ae ee i bit laa nea fs Re ae ¢ Similar final stats 
= Taha 
: » a ae Hit i , 
S H mar | i I facia en ; + 
S | mn aid Tai arias i: +3 % error 
o (junta y i i ; 
-0.04 : 
-0.06 
-0.08 


Run Number 


Model confirmed by validation test; 7 points fail 3% error test 
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Validation Test, Blended DOE-L2F aa 


¢ Residuals vs Run 
ied ¢ Blocks 1-4 
¢ Cy low arange 
¢ Case #3 model 
Similar final stats 


0.05 — 


+3 % error 


Residuals 


1 432 863 1294 


Run Number 


Model confirmed by validation test; 6 points fail 3% error test 
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Concluding Remarks 


¢ Sequential testing & validation recommended 
— Obtain data sequentially as required 
— Apply validation test after each block of data 


¢ Efficient test methods demonstrated 
— DOE & L2F approaches provide methods to increase efficiency 
— Blending DOE-L2F 
¢ Currently a “work in progress” but shows promise 
¢ Presents a challenge in design phase to combine LBW+HBW factors 
¢ Future Test Refinements 
— Fewer blocks required with continuous sampling 
— Smaller optimal blocks 
— Lower sample rates for “quasi-static” tests 
— For “quasi-static” case, lower bandwidth of HBW inputs 
— Design must reflect significant data added by HBW factors 
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Questions? asa 


¢ Contact Information 


— patrick.c.murphy@nasa.gov ” 
but some are useful” — 
— /5/-864-4071 


| George E. P. Box 
— jay.m.brandon@nasa.gov 
— 757-864-1142 


“All models are wrong, 


Replication 


Orthogonality Randomization 


Blocking 
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